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About  the  Technical  Note  Series  on  Architecture  Evaluation  in 
the  Department  of  Defense 


The  Product  Line  Systems  Program  is  publishing  a  series  of  technical  notes  designed  to  con¬ 
dense  knowledge  about  architecture  evaluation  practices  into  a  concise  and  usable  form  for 
the  Department  of  Defense  (DoD)  acquisition  manager  and  practitioner.  This  series  is  a  com¬ 
panion  to  the  Software  Engineering  Institute  (SEI)  series  on  product  line  acquisition  and 
business  practices  [Bergey  99]. 

Each  technical  note  in  the  series  will  focus  on  the  use  of  architecture  evaluation  and,  in  par¬ 
ticular.  on  applying  the  SEI’s  architecture  tradeoff  analysis  technology  in  the  Department  of 
Defense.  Our  objective  is  to  provide  practical  guidance  on  ways  to  integrate  sound  architec¬ 
ture  evaluation  practices  into  their  acquisitions.  This  series  of  technical  notes  will  lay  down  a 
conceptual  foundation  for  DoD  architecture  evaluation  practice. 
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Abstract 


The  software  architecture  of  a  system  is  a  major  determinant  of  software  quality  and  one  of 
the  earliest  artifacts  available  for  evaluation.  For  a  government  acquisition  organization,  the 
ability  to  evaluate  software  architectures  can  have  a  favorable  impact  on  the  delivered  sys¬ 
tem.  This  technical  note  describes  the  application  of  the  Architecture  Tradeoff  Analysis 
MethodSM  (ATAMsm)  to  evaluate  a  reference  architecture  for  ground-based  command  and 
control  systems.  The  use  of  the  term  reference  architecture  in  the  context  of  this  application 
is  presented.  A  general  overview  of  the  ATAM  process  is  provided  and  the  results  of  the 
ATAM  are  explored,  including  the  benefits  of  performing  an  ATAM-based  architecture 
evaluation  both  to  the  acquirer  and  to  the  developer. 


SM  Architecture  Tradeoff  Analysis  Method  and  ATAM  are  service  marks  of  Carnegie  Mellon 
University. 
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1  Introduction 


Most  modem  defense  systems  rely  heavily  on  software  to  achieve  system  functionality.  Be¬ 
cause  software  architecture  is  a  major  determinant  of  software  quality,  it  follows  that  soft¬ 
ware  architecture  is  critical  to  the  quality  of  a  software-intensive  system.  For  a  Department  of 
Defense  (DoD)  acquisition  organization,  the  ability  to  evaluate  software  architectures  can 
reduce  the  risk  that  the  delivered  system  will  not  meet  its  quality  goals. 

This  technical  note  describes  the  application  of  the  Architecture  Tradeoff  Analysis  Method 
(ATAM)  in  the  evaluation  of  a  government-sponsored  reference  architecture  for  a  ground- 
based  command  and  control  system.  The  names  of  the  organizations  involved  and  the  refer¬ 
ence  architecture  are  not  provided  since  they  are  not  necessary  for  the  discussion.  The  use  of 
the  term  reference  architecture  is  explored  as  well  as  how  it  relates  to  a  product  line  archi¬ 
tecture.  A  general  overview  of  the  ATAM  process  is  provided  and  the  results  of  the  ATAM  are 
explored,  including  the  benefits  of  performing  an  ATAM-based  architecture  evaluation  both 
to  the  acquirer  and  the  developer. 


CMU/SEI-2000-TN-007 


1 


2  The  Context  for  the  Architecture  Evaluation 


The  Architecture  Tradeoff  Analysis  Method  was  used  to  evaluate  a  reference  architecture  for 
ground-based  command  and  control  systems.  To  explore  this  application  of  ATAM,  we  first 
need  to  define  the  terms  being  used  and  describe  the  evaluation  context.  We  begin  with  the 
definition  of  software  architecture. 

2.1  What  is  Software  Architecture? 

The  software  architecture  of  a  program  or  computing  system  is  the  structure 
or  structures  of  the  system,  which  comprise  software  components,  the  exter¬ 
nally  visible  properties  of  those  components,  and  the  relationships  among 
them  [Bass  98]. 

The  software  architecture  represents  the  earliest  software  design  decisions.  These  design  de¬ 
cisions  are  the  most  critical  to  get  right  and  the  most  difficult  to  change  downstream  in  the 
system  development  life  cycle.  The  software  architecture  is  the  first  design  artifact  addressing 
reliability,  modifiability,  security,  real-time  performance,  and  interoperability  goals  and  re¬ 
quirements. 

There  are  many  different  views  of  a  software  architecture,  and  which  one  is  relevant  depends 
on  the  stakeholders  and  the  system  properties  that  are  of  interest.  If  we  consider  the  analogy 
of  the  architecture  of  a  building,  various  stakeholders  such  as  the  construction  engineer,  the 
plumber,  and  the  electrician  all  have  an  interest  in  how  the  building  is  to  be  constructed. 
However,  they  are  interested  in  different  structural  units  and  different  structural  relationships; 
they  each  have  a  different  view  of  what  constitutes  the  architecture.  Each  of  their  views  of 
architecture  is  valid.  Each  represents  a  structure  that  maps  to  one  of  the  structural  goals  of  the 
building,  and  all  views  are  necessary  to  represent  the  architecture  of  the  building  fully.  Simi¬ 
larly,  a  software  architecture  has  a  variety  of  stakeholders,  including  possibly  the  develop¬ 
ment  organization,  the  end  user,  the  system  maintainer,  the  operator,  and  the  acquisition  or¬ 
ganization.  Each  of  these  stakeholders  has  a  vested  interest  in  different  system  properties  and 
goals  that  are  structurally  represented  by  different  views  of  the  system.  These  different  prop¬ 
erties  and  goals  and  their  corresponding  architectural  views  are  important  to  understand  and 
to  analyze  in  order  to  reason  about  the  appropriateness  and  the  quality  of  the  architecture. 

The  software  architecture  to  be  evaluated  was  actually  a  reference  architecture. 
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2.2  What  is  a  Reference  Architecture? 


A  reference  architecture  is  the  generalized  architecture  of  several  end  systems  that  share  one 
or  more  common  domains.  The  reference  architecture  defines  the  infrastructure  common  to 
the  end  systems  and  the  interfaces  of  components  that  will  be  included  in  the  end  systems. 

The  reference  architecture  is  then  instantiated  to  create  a  software  architecture  of  a  specific 
system.  The  definition  of  the  reference  architecture  facilitates  deriving  and  extending  new 
software  architectures  for  classes  of  systems.  A  reference  architecture,  therefore,  plays  a  dual 
role  with  regard  to  specific  target  software  architectures.  First,  it  generalizes  and  extracts 
common  functions  and  configurations.  Second,  it  provides  a  base  for  instantiating  target  sys¬ 
tems  that  use  that  common  base  more  reliably  and  cost  effectively. 

The  concept  of  a  reference  architecture  is  very  similar  to  that  of  an  application  framework  in 
the  parlance  of  object  technologists  and  to  that  of  a  product  line  architecture.  In  fact,  the 
terms  reference  architecture  and  product  line  architecture  are  often  used  interchangeably.  A 
product  line  architecture  is  the  architecture  for  a  software  product  line.1 

Within  a  government  context,  the  establishment  of  a  reference  architecture  that  can  span  a 
broad  range  of  missions,  development  organizations,  and  operations  concepts  is  motivated  by 
the  desire  of  end-system  customers  and  integrators.  These  groups  wish  to  achieve  high  levels 
of  strategic  software  reuse  and  interoperability.  The  reference  architecture  provides  specifi¬ 
cations  for  both  system  and  component  developers. 

2.3  The  Goals  of  the  Architecture  in  Question 

The  reference  architecture  being  evaluated  was  not  developed  to  support  a  software  product 
line,  but  rather  to  provide  a  common  framework  for  classes  of  ground-based  command  and 
control  systems  being  built  by  multiple  government  and  commercial  organizations. 

The  primary  goal  of  this  reference  architecture  is  to  standardize  the  interfaces  between  soft¬ 
ware  components  used  in  ground-based  command  and  control  systems.  The  expectation  is 
that  the  standardized  interfaces  will  reduce  the  risk  and  cost  of  implementing  new  systems  by 
facilitating  the  use  of  standard  components  in  a  “plug-and-play”  mode.  The  stated  vision  is 
that  vendors  will  develop  components  that  are  compliant  with  the  reference  architecture, 
making  the  job  of  a  system  integrator  easier,  quicker,  and  less  risky.  Figure  1  illustrates  this 
vision. 


1  A  software  product  line  is  a  set  of  software-intensive  systems  sharing  a  common,  managed  set  of 
features  that  satisfy  the  specific  needs  of  a  particular  market  segment  or  mission  and  that  are 
developed  from  a  common  set  of  core  assets  in  a  prescribed  way. 
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Figure  1:  The  Intended  Use  of  the  Reference  Architecture  Being  Evaluated 


2.4  The  Acquisition  Organization 

Because  the  reference  architecture  for  ground-based  command  and  control  systems  is  envi¬ 
sioned  to  be  a  common  starting  point  across  many  instantiations  of  systems,  acquisition  or¬ 
ganizations  from  two  distinct  government  agencies  provided  funding. 

Acquisition  management  responsibilities  were  split  between  the  two  major  agencies,  and  the 
business  case  and  generalized  requirements  for  the  reference  architecture  were  developed  in 
conjunction  with  both  communities. 


2.5  The  Development  Organization 

The  two  agencies  used  various  methods  to  encourage  developers  to  participate  in  the  defini¬ 
tion  of  the  reference  architecture.  Some  developers  participated  as  part  of  an  ongoing  support 
role  to  an  agency,  while  others  participated  as  a  task  on  an  existing  development  effort  funded 
by  the  agency.  In  addition,  vendors  interested  in  providing  components  compliant  with  the 
reference  architecture  supplied  developers  to  participate  in  the  definition  of  the  architecture 
using  internal  funds. 
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3  The  ATAM 


3.1  Purpose 

The  purpose  of  the  ATAM  is  to  assess  the  consequences  of  architectural  decision  alternatives 
in  light  of  quality  attributes.  The  method  ensures  the  right  questions  are  asked  early  to  dis¬ 
cover 

•  risks:  alternatives  that  might  create  future  problems  in  some  quality  attribute 

•  sensitivity  points:  alternatives  for  which  a  slight  change  makes  a  significant  difference  in 
a  quality  attribute 

•  tradeoffs:  decisions  affecting  more  than  one  quality  attribute  [Kazman  98]2 

The  ATAM  is  intended  to  analyze  an  architecture  with  respect  to  its  quality  attributes,  not  its 
functional  correctness. 

The  ATAM  involves  a  wide  group  of  stakeholders  (including  managers,  developers,  main¬ 
tained,  testers,  reusers,  end  users,  and  customers)  in  an  effort  to  surface  the  relevant 
stakeholders’  quality  goals  for  the  system.  The  ATAM  is  a  method  for  mitigating  architecture 
risk,  a  means  of  detecting  areas  of  potential  risk  within  the  architecture  of  a  complex  soft¬ 
ware-intensive  system,  and  not  a  precise  mathematical  analysis.  As  such,  the  ATAM  can  be 
done  early  in  the  software  development  life  cycle,  and  it  can  be  done  inexpensively  and 
quickly. 

It  does  not  need  to  produce  detailed  analyses  of  any  measurable  quality  attribute  of  a  system 
(such  as  latency  or  mean  time  to  failure)  to  be  successful,  but  it  instead  identifies  trends 
where  some  architectural  parameter  is  correlated  with  a  measurable  quality  attribute  of  inter¬ 
est. 


3.2  ATAM  Steps 

The  ATAM  process  consists  of  the  following  10  steps: 

1 .  Present  the  ATAM:  a  quick  overview  by  the  evaluation  team  of  the  ATAM  steps,  tech¬ 
niques  used,  and  outputs  from  the  process. 

2.  Present  the  business  drivers:  a  brief  presentation  by  the  system  manager  describing  the 
business  drivers  and  context  for  the  architecture. 

3.  Present  the  architecture:  the  architect’s  presentation  of  the  architecture. 


2  The  ATAM  has  been  improved  since  the  application  described  in  this  report  [Kazman  00]. 
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4.  Identify  architectural  styles:  an  itemization  of  styles  discovered  as  a  result  of  the  previ¬ 
ous  step. 

5.  Generate  the  quality  attribute  utility  tree:  identification,  prioritization,  and  refinement  of 
the  most  important  quality  attribute  goals  in  the  form  of  a  utility  tree. 

6.  Elicit  and  analyze  architectural  styles:  a  probing  of  the  architectural  styles  in  light  of  the 
quality  attributes  in  order  to  identify  risks,  sensitivity  points,  and  tradeoffs. 

7.  Generate  seed  scenarios:  a  representation  of  the  stakeholder’s  interest  to  understand 
quality  attribute  requirements. 

8.  Brainstorm  and  prioritize  scenarios:  addition  of  scenarios  from  stakeholders  and  an  un¬ 
derstanding  of  their  relative  importance. 

9.  Map  scenarios  onto  styles:  continuing  to  identify  risks,  sensitivity  points,  and  tradeoffs 
while  noting  styles  and  components  within  styles  that  are  affected  by  each  scenario. 

10.  Present  out-brief  and/or  write  report:  recapitulation  of  the  execution  of  the  ATAM  steps, 
results,  and  recommendations. 
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4  The  Evaluation 


4.1  Timing  of  the  Evaluation 

At  the  time  of  the  ATAM  evaluation,  the  development  team  for  the  reference  architecture  had 
progressed  through  the  following  three  phases: 

1 .  Understand  the  problem  and  scope  the  effort. 

2.  Document  the  reference  architecture  and  develop  high-level  Interface  Design  Language 
(IDL). 

3.  Complete  approximately  one-fourth  of  the  IDL  for  the  12  sub-domains. 

4.2  Phase  1  of  the  Evaluation 

The  ATAM  method  is  best  applied  in  two  phases.  The  first  phase  involves  the  initial  connec¬ 
tion  of  the  evaluation  team  leads  with  the  architects  and  the  beginning  of  the  exploration  and 
analysis.  Phase  1  allows  the  evaluation  team  to  get  acquainted  with  the  architect(s),  the  sys¬ 
tem  purpose,  and  the  architecture  and  to  begin  a  preliminary  analysis  of  the  architecture. 
Phase  1  also  permits  the  architects  to  become  familiar  with  the  ATAM  and  understand  the 
information  they  must  supply  for  the  complete  evaluation  to  proceed. 

An  SEI  ATAM  evaluation  team  was  selected  for  this  evaluation.  The  team  leads  met  with  ref¬ 
erence  architecture  architects  and  provided  templates  for  architecture  presentation  to  be  given 
as  part  of  the  evaluation  exercise.  A  draft  utility  tree  and  seed  scenarios  were  produced.  A 
template  was  also  given  to  the  architecture  manager  to  be  used  in  the  presentation  of  the 
business  drivers. 

4.3  Phase  2  of  the  Evaluation 

All  of  the  ATAM  steps  are  executed  during  Phase  2,  which  typically  lasts  two  days.  Any  pre¬ 
liminary  analysis  resulting  from  Phase  1  is  presented. 

For  this  evaluation,  an  assembly  of  approximately  two  dozen  stakeholders  gathered  for  the 
two-day  event.3  These  stakeholders  represented  the  two  government  acquisition  organizations 
as  well  as  a  variety  of  government  contractors  with  a  vested  interest  in  the  reference  archi¬ 
tecture.  Some  of  these  contractors  had  been  involved  with  the  definition  of  the  reference  ar¬ 
chitecture.  Some  hoped  to  build  components  that  would  be  compliant  with  the  reference  ar¬ 
chitecture. 


3  The  typical  use  of  ATAM  involves  a  much  smaller  group  of  stakeholders.  In  this  case,  the  two 
government  acquisition  agencies  felt  it  was  necessary  to  involve  a  broader  set. 
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All  steps  of  the  ATAM  were  executed  during  the  two-day  period.  The  architects  used  the 
templates  provided  in  their  presentation  of  the  architecture.  The  draft  utility  tree  was  pre¬ 
sented  and  then  refined  by  the  larger  convened  group.  Scenarios  were  generated  and  priori¬ 
tized  by  the  entire  group.  Some  key  aspects  of  this  evaluation  were  the  business  drivers  and 
the  architectural  artifacts  that  were  available. 

4.3.1  Business  Drivers 

One  of  the  most  important  steps  when  performing  an  ATAM-based  architecture  evaluation  is 
to  elicit  from  the  acquirer  a  clear  articulation  of  the  business  case  driving  the  development  of 
the  architecture.  During  this  evaluation,  a  representative  from  one  of  the  acquisition  agencies 
described  the  business  goals  motivating  the  development  effort  and  hence  identified  the  pri¬ 
mary  drivers  (e.g.,  high  availability,  time  to  market,  or  high  security)  for  the  architecture.  The 
presentation  of  the  business  goals  included  these  key  points: 

•  The  goal  of  the  reference  architecture  is  to  enable  system  builders  to  integrate  systems  on 
time,  within  costs,  while  meeting  performance  needs;  the  primary  objective  is  to  drive 
down  the  cost  of  command  and  control  systems.  The  strategy  involves  the  use  of  object- 
oriented  technology  and  Common  Object  Request  Broker  Architecture  (CORB  A)  to 
achieve  this  goal. 

•  The  focus  is  on  defining  interfaces  to  sub-domains  with  an  emphasis  on  interoperability 
among  the  sub-domains  as  opposed  to  reusing  sub-domains  directly. 

•  The  definition  of  the  standard  interfaces  gives  vendors  a  target  and  permits  competition 
among  vendors  in  developing  products  for  the  same  sub-domain. 

•  System  builders  should  be  able  to  use  the  reference  architecture  to  integrate  the  sub- 
domains  quickly.  Under  current  practices,  integration  costs  are  very  high.  Point  solutions 
are  too  expansive  and  too  late. 

•  Customers  want  open  systems. 

•  The  reference  architecture  will  lower  the  cost  and  cycle  time  for  system  integrators. 

•  The  reference  architecture  will  help  meet  future  challenges  by  enabling 

-  constellations  of  systems 

-  “plug  and  play,”  regardless  of  location  of  component  (autonomous  versus  under  direct 
control) 

The  final  point  was  aimed  at  being  able  to  demonstrate  the  “goodness”  of  the  reference  ar¬ 
chitecture  to  other  government  agencies. 

4.3.2  Architectural  Artifacts  Available  for  the  Evaluation 

The  artifacts  available  to  describe  the  ground-based  command  and  control  reference  archi¬ 
tecture  included  of  a  set  of  definitions  for  large-grained  components,  IDL  interfaces  for  these 
components,  and  CORB  A  mechanisms  used  to  connect  the  components.  A  number  of  use- 
cases  were  also  presented  during  the  ATAM  evaluation,  and  their  realization  in  the  architec¬ 
ture  was  briefly  discussed.  As  noted  earlier,  the  reference  architecture  specification  was  not 
complete  at  the  time  of  the  evaluation. 
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5  Evaluation  Results 


5.1  The  Findings 

The  2-day  ATAM  evaluation  produced  a  list  of  22  risks,  which  were  grouped  into  the  fol¬ 
lowing  6  risk  themes: 

1 .  The  architecture  provides  no  mechanisms  for  the  integrator  to  reason  about  quality  at¬ 
tribute  tradeoff  (for  example,  real-time  performance  versus  reliability). 

2.  The  definition  of  the  reference  architecture  did  not  include  key  stakeholder  requirements 
for  real-time  performance,  integrability,  modifiability,  and  availability. 

3.  There  are  major  tradeoffs  between  affordability,  real-time  performance,  and  modifiabil¬ 
ity  that  have  been  explicitly  left  to  the  integrator  to  address. 

4.  There  are  no  overarching  quality  attribute  models  (for  example,  real-time  performance 
models,  availability  models). 

5.  There  is  a  lack  of  common  support  for  managing  data. 

6.  The  direct  reliance  on  CORBA  as  a  distributed  system  poses  a  risk  for  CORBA  evolu¬ 
tion  or  change  to  a  different  middleware. 

Based  upon  the  ATAM  evaluation,  the  evaluation  team  concluded  that  little  had  been  done  in 
the  definition  of  the  reference  architecture  to  address  the  quality  goals  of  the  ground-based 
command  and  control  systems  to  be  supported  by  this  architecture.  The  fulfillment  of  those 
goals  would  be  left  to  the  integrators,  leaving  the  likely  result  that  components  built  to  be 
compliant  with  the  reference  architecture  would  indeed  “plug,”  with  no  real  guarantee  that 
they  would  “play.”  In  other  words,  the  business  drivers  for  the  reference  architecture  are  at 
risk  with  the  given  architecture  definition. 

5.2  The  Benefits 

An  architecture  helps  the  developer  and  customer  reason  about  a  proposed  system  during  its 
development  and  evolution.  A  structured  approach  to  eliciting  that  reasoning  early  in  devel¬ 
opment  increases  the  system  developer’s  probability  that  a  system  built  conforming  to  the 
architecture  will  meet  the  needs  of  its  customer  base. 

The  two  days  spent  using  the  ATAM  to  analyze  the  reference  architecture  for  ground-based 
command  and  control  systems  pointed  out  potential  deficiencies  that  may  have  taken  months, 
perhaps  years,  to  uncover  at  a  greatly  increased  cost  to  the  acquirer. 

The  ATAM  evaluates  an  architecture  with  respect  to  its  quality  goals.  In  this  case,  the  goals 
were  performance,  security,  availability  and  modifiability.  It  became  evident  fairly  quickly 
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that  the  proposed  architecture  did  not  provide  any  architectural  mechanisms  to  aid  the  inte¬ 
grator  in  ensuring  that  the  intended  quality  goals  were  met. 

The  team  of  developers  and  acquirers  of  the  reference  architecture  came  away  from  the  two- 
day  ATAM  with  a  clear  understanding  that  more  work  needed  to  be  done  to  ensure  that  not 
only  were  the  interfaces  syntactically  correct,  but  that  the  components  were  integrated  se¬ 
mantically  as  well. 

The  benefits  of  performing  an  ATAM  are  clear:  early  identification  of  risks,  sensitivity  points, 
and  tradeoffs  before  design  decisions  are  made  and  become  costly  to  change.  As  one  partici¬ 
pant  from  the  government  team  noted  after  the  results  of  the  ATAM  were  briefed,  “Why 
wouldn’t  a  program  want  to  do  an  ATAM-based  evaluation?” 
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6  Conclusion 


This  note  has  described  the  application  of  the  ATAM  to  conduct  an  architecture  evaluation 
during  the  development  of  a  government-sponsored  reference  architecture  for  ground-based 
command  and  control  systems.  The  context  for  the  evaluation  was  given.  A  general  overview 
of  the  ATAM  process  was  presented,  and  finally,  the  results  of  this  ATAM-based  evaluation 
were  examined  as  well  as  the  perceived  benefits  to  both  the  acquirer  and  the  developer  for 
performing  the  evaluation. 

The  SEI  is  collaborating  with  several  government  acquisition  organizations  to  explore  the 
appropriate  use  of  the  ATAM  within  these  organizations,  to  help  them  adopt  architecture 
evaluation  practices,  and  to  help  them  include  the  appropriate  language  in  a  request  for  pro¬ 
posal  (RFP)  to  ensure  that  architecture  evaluation  is  an  integral  part  of  evaluating  proposals. 
As  experience  is  gained,  we  will  continue  to  share  our  lessons  learned  in  future  technical 
notes  in  this  series. 
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Feedback  and  Contact 


Comments  or  suggestions  about  this  document  or  the  series  of  technical  notes  on  architecture 
evaluation  in  the  Department  of  Defense  are  welcome.  We  want  this  series  to  be  responsive  to 
the  needs  of  DoD  and  government  personnel.  To  that  end,  comments  concerning  this  techni¬ 
cal  note,  inclusion  of  other  topics,  or  any  other  issues  or  concerns  will  be  of  great  value  in 
continuing  this  series.  Comments  or  suggestions  should  be  sent  to 

Linda  Northrop,  Director 
Product  Line  Systems  Program 
lmn@sei.cmu.edu 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 
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